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Summary  and  Highlights 


Problem 

There  are  23  state  director  positions  in  the  Federal  Crop  Insurance 
Corporation,  all  classified  at  essentially  the  same  level.  The  range  of 
complexities  and  responsibilities  among  them  is  such  as  to  lead  FCIC  top 
management  to  seek  some  obiective  basis  for  differential  classification. 
Previous  efforts  to  agree  on  variables  upon  which  a fair,  objective,  and 
operationally  useful  set  of  standards  could  be  developed  had  ended  in 
frustrationo 

Method 

The  problem  was  approached  through  simulation,  and  analysis  of  the 
decision-making  process  employed  by  9 top  level  FCIC  managers  in  attempt- 
ing to  develop  classification  standards » The  following  steps  were  involved, 

la  The  managers  discussed  the  question;  ”What  factors  ought  to  be 
used  in  developing  the  desired  estimates?”  Eleven  factors  were  agreed  upon, 

2a  A set  of  23  data  sheets  was  prepared.  Each  sheet  contained  infor- 
mation on  the  11  factors  for  a single  state  director  position o 

3,  Each  of  the  9 managers  sorted  the  23  sheets  into  3 piles  on  the 
basis  of  his  individual  judgment  (Highest  7,  Medium  9,  Lowest  7),  This  was 
done  twice — once  on  the  basis  of  the  relative  level  of  the  positions  without 
regard  to  the  performance  of  the  incumbent;  once  taking  the  performance  of 
the  incumbent  into  account , 

Results 

1,  The  level  of  agreement  among  members  of  the  judging  group  was 
satisfactory, 

2,  The  ranking  of  the  positions  was  not  changed  in  any  significant 
degree  by  taking  the  present  incumbent’s  performance  into  account, 

3o  Analytical  reconstruction  of  the  decision-making  process  used  by 
the  9 managers,  showed  that  2 information  factors  could  be  used  that  would 
give  the  same  results  as  all  11,  These  were; 

(a)  Ratio  of  total  acres  insured  in  the  state  in  existing  counties 
to  total  insurable  acres  in  existing  counties  (’’existing” 
counties  are  those  in  which  crop  insurance  programs  are  cur- 
rently established); 

(b)  Dollar  amount  of  premium  income. 

These  appear  to  be  the  factors  most  feasible  to  use  in  attempting  to 
develop  standards  for  differential  classification  of  FCIC  state  director 
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Introduction 


The  top  management  group  of  the  Federal  Crop  Insurance  Corporation 
(FCIC)  asked  the  Personnel  Research  Staff  (PRS)  to  conduct  a study  of  their 
process  of  estiTnacing  the  relative  level  of  FCIC  state  director  positions b 

This  request  grew  out  of  a description  by  FCIC  management  of  a wide 
range  of  complexity  and  responsibilities  among  the  23  state  director 
positionS“«the  top  field  positions  in  FCIC==and  an  expressed  need  to  iden« 
tify  standards  that  might  provide  some  objective  basis  for  differential 
classification  of  these  posit ions o At  present  they  are  all  classified  at 
the  same  level <> 

Previous  attempts  to  agree  on  variables  upon  which  a fair^  objective 
and  operationally  useful  set  of  standards  could  be  developed  had  ended  in 
frustrationo 

It  was  thought  that  some  clarification  of  the  problem  involved  might 
be  obtained  by  simulation  and  analysis  of  the  decision-making  process  FCIC 
top  management  employed  in  attempting  to  develop  classification  standards o 

The  following  report  presents  the  procedural  steps  involved  and  the 
empirical  results  of  this  studyo 

The  Factors 

Several  sessions  were  held  in  which  the  group  discussed  the  question? 
"What  are  the  factors  that  ought  to  be  used  in  developing  the  desired 
estimates?"  Some  of  the  factors  considered  appeared  to  be  nearly  identical 
or  were  composites  of  ether  factors o For  other  factors 4 information  on  all 
state  director  positions  could  not  be  readily  obtained 0 A consenius  was 
finally  achieved  that  information  about  the  23  state  director  positions 
would  be  developed  on  the  following  eleven  factors? 

lo  Ratio  of  total  acres  insured  in  the  state  in  "existing"  counties 
to  total  insurable  acres  in  "existing"  counties  ("existing" 
counties  are  those  in  which  crop  insurance  programs  are  currently 
established) 

2o  Dollar  amount  of  premium  income 

3o  Amount  of  insured  liability 

4o  Number  of  crops  insured 

5o  Number  of  crop  cancellations 

60  Number  of  counties  with  programs 
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7o  Number  of  crops  earning  premiums 
8o  Number  of  contracts  in  force 
9o  Number  of  potential  insurable  crops 
10 o Number  of  insurance  units  insured 
lie  Number  of  different  commodities  insured 


The  Data 

Information  on  the  above  eleven  factors  was  obtained  from  FCIC  records 
for  each  of  the  23  state  director  posit ions o A set  of  data  sheets  was  then 
prepared  with  each  sheet  containing  the  information  on  the  eleven  factors 
for  a single  state  director  positiono 

Each  of  the  nine  top  level  managers  of  FCIC  who  participated  in  this 
phase  of  the  study  was  given  two  sets  of  the  twenty^three  information 
sheet So  Each  participant  was  told  to  use  the  information  provided  to  sort 
each  set  into  three  piles  on  the  basis  of  his  individual  judgment,  as 
follows s 


Statje  Director  Position  Level 

Hi  gh<rst~”^"Med  i'lan  Lowest 

7'""  ' 9 ’7 

The  participants  were  not  allowed  to  communicate  with  one  another 
during  the  sorting  procedure o 

The  first  set  was  sorted  on  the  basis  of  the  relative  level  of  the 
positions  as  currently  constituted  without  regard  to  the  performance  of 
the  incumbent o The  second  set  was  sorted  on  the  basis  of  the  relative 
level  of  the  position  as  performed  by  the  present  incumbent o The  verbatim 
instructions  are  given  in  Appendix  Ao  The  positions  were  not  identified 
by  state  for  the  first  sorting,  but  were  for  the  second  sortingo 

The  two  separate  sortings  were  used  to  deal  with  the  questions  raised 
in  the  earlier  discussions?  **Should  the  performance  of  present  incumbents 
be  considered  in  deciding  upon  the  level  at  which  to  place  the  positions? 
Would  it  make  a difference  in  the  way  in  which  the  23  positions  were 
ranked  by  the  judges?” 

To  simplify  further  reference  we  will  hereafter  identify  the  first 
"level  alone”  sorting  as  L,  and  the  sorting  that  included  performance 
considerations  as  L^-Po 

A twelfth  information  item  that  was  obtained  for  use  in  later 
analyses,  but  not  for  use  by  the  judges,  was  the  length  of  time  each 
state  director  had  served  in  that  capacityo  This  variable  was  included 
to  see  whether  it  had  any  influence  upon  the  judgments  of  the  participants o 
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The  Analyses  and  Results 

Agreement  among  the  participantSo  Each  position  was  scored  9 times ^ 
one  score  :^rom  each  judge,  on  the  basis  of  the  sorting  process  described 
above;  Highest  2,  Medium  1,  and  Lowest  0*  The  sortings  were  first 
analyzed  to  determine  the  extent  of  agreement  among  members  of  the  judging 
group o 

The  intraclass  correlation  for  ^ sortings  was  *62.  For  LtP  it  was 
.73.  (loOO  would  represent  perfect  agreement;  OoOO  complete  absence  of 
agreement o)  The  computational  rationale  for  this  index  and  the  variance 
components  are  given  in  Appendix  B, 

For  data  of  this  kind,  when  each  person  must  assign  a specified 
number  of  sheets  to  each  pile,  intraclass  correlations  in  this  range  are 
quite  adequate  and  indicate  a fairly  high  degree  of  agreement  among  the 
group  members o This  made  it  legitimate  and  desirable  to  combine  the  nine 
judgments  into  one  average  score  for  each  of  the  state  director  positions o 

Correlation  of  L with  LtP  score So  The  average  L and  LtP  scores  were 
then  correlated e The  correlation  obtained  was  609  which,  taking  into 
account  the  reliability  of  the  original  judgments,  indicated  that  the 
ordering  of  the  positions  was  almost  identical  for  these  two  sets  of 
pooled  judgments* 

Even  though  these  differences  were  slight  it  was  decided,  as  a fxirther 
check,  to  separately  relate  the  information  factors  to  the  average  L and 
LtP  scores*  It  was  thought  that  the  L scores  might  be  better  reproduced 
by  the  factors  than  would  the  LtP  scores,  since  information  not  contained 
in  the  factors  might  be  taken  into  account  in  jua^ing  performance  * The 
factor  of  "time  served  in  position"  was  introduced  into  the  analysis  at 
this  point* 

The  intercorrelations  among  the  factors  are  given  in  Appendix  C* 

Factors  influencing  the  decision  process*  "What  factors  most  in» 
fluenced  the  decisions  of  our  FCIC  top=>level  executives  as  to  the  order 
in  which  to  place  the  state  director  positions?"  Posed  in  analytic  terras 
the  question  becomes:  "How  close  can  we  come  to  reproducing  the  ranking 
of  the  23  state  director  positions  obtained  from  the  average  score  assigned 
by  the  9 judges,  using  the  12  information  factors?"  This  may  be  thought 
of  as  an  anal3rtical  reconstruction  of  the  decision=making  process  in  which 
the  "average  score  rank"  is  regarded  as  the  best  available  overall  decision* 

The  technique  employed  is  known  as  "multiple  regression  analysis*" 

The  manner  in  which  this  technique  utilizes  information  is  explained  in 
Appendix  D*  How  well  this  technique  reproduces  the  pooled  judgments  is 
expressed  by  a squared  multiple  correlation  coefficient  (SMC)  that  ranges 
from  0*00  for  absence  of  any  accurate  reproduction  to  1*00  for  completely 
accurate  reproduction  of  the  average  score  rank* 
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Influence  of  factors  upon  decisiono  The  first  question  to  be  answered 
iss  "Does  length  of  time  served  in  the  state  director’s  position  contri- 
bute any  informations  not  already  supplied  by  the  other  factors*  in  repro- 
ducing the  average  L or  LtP  scores'?*  Exhibit  1 shows  the  SMC  obtained  when 
**length  of  time  served**  was  included  and  when  it  was  excluded  from  the 
analysis  with  the  eleven  other  factorso  These  figures  show  that  inclusion 
of  **length  of  time  served**  did  not  make  any  appreciable  difference  in  the 
results o 


Exhibit  1 

Squared  Multiple  Correlation  When  **Length  of  Time  Served** 
Is  Included  and  Excluded  from  the  Analysis 


Level 

Performance 

Diff, 

Included 

SMC  = ,90 

SMC  - ,86 

o 

o 

Excluded 

SMC  = o90 

SMC  = ,85 

,05 

Difference 

oOO 

O 

o 

Exhibit  1 also  shows  that  there  is  not  any  practical  difference  between 
the  SMCs  for  L and  for  LtPo  They  can  both  be  reproduced  with  a very  high 
degree  of  acciTracyo 

The  next  question  to  be  answered  is 3 "Which  of  the  eleven  factors 
best  reproduce  the  ^ and  LtP  scores?"  The  SMCs  for  L and  LtP  were  o88  and 
<>85  respectively*  using  the  **best**  four  factors  in  each  instance o These 
four  factors  v;ere  1*  2*  3 and  6 for  L*  and  1*  2*  7 and  9 for  LtPo  (See 
the  list  of  factors*  page  lo) 

Since  there  is  no  appreciable  difference  in  the  accuracy  with  which 
these  two  sets  of  scores  can  be  reproduced  and  since  the  two  sets  of  scores 
were  shown  to  be  very  highly  correlated  (o89)*  the  slight  differences  that 
do  exist  between  them  can  be  attributed  to  the  disagreements  among  the 
members  of  the  group  in  arriving  at  their  decisions o If  there  was  a greater 
degree  of  agreement  among  the  group  members  in  arriving  at  their  decisions 
the  correlation  between  the  two  sets  of  scores  would  be  even  higher.  Since 
the  and  LtP  scores  could  be  regarded  as  being  essentially  similar*  the 
next  question  wass  ’*How  accurately  could  factors  1 and  2*  which  were  among 
the  four  **best**  for  both  sets  of  scores*  reproduce  the  average  scores?** 

The  L score  was  considered  to  be  the  conceptually  clearer  of  the  two.  When 
this  analysis  was  performed  the  SMC  obtained  v/ith  L was  ,83, 
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This  appears  close  enough  to  the  maximum  obtainable  with  all  factors 
to  suggest  that  the  use  of  these  two  factors  alone  would  suffice  for  all 
practical  purposes s 

(a)  Ratio  of  total  acres  insured  in  the  state  in  existing  counties 
to  total  insurable  acres  in  existing  counties 

(b)  ^Dollar  amount  of  premium  income 

Use  of  additional  factors,  which  would  have  relatively  minor  weights, 
would  require  more  work  without  making  any  aporeciable  difference  in  the 
decision  arrived  ato 

To  combine  these  factors  they  should  first  be  ranked  and  added  ud  by 
giving  (a;  a weight  of  1,  and  (b)  a weight  of  2 (i^eo  2(b)  + 1(a)  = Weighted 
Rank)o  A comparison  of  the  ranks  obtained  on  the  basis  of  these  two  factors 
and  the  average  score  rankings  is  given  in  Exhibit  2„ 

Inspection  of  Exhibit  2 shows  that  the  six  states  highest  on  "weighted 
rank"  are  the  same  six  to  which  the  judges  assigned  highest  "average  score 
ranks"  (d,  j,  p,  r,  s,  u).  Similarly  the  six  lowest  on  "weighted  rank"  are 
among  the  seven  lowest  in  "average  score  rank"  (b,  e,  g,  i,  1,  n)« 

Conclusion 


If  one  accepts  the  premise  that  the  pooled  judgment  of  the  relative 
level  of  state  director  positions,  made  by  the  nine  FCIC  executives  who 
participated  in  this  simulated  decision-making  process,  represents  an 
operationally  valid  criterion,  then  it  would  apnear  that  they  could  arrive 
at  practically  the  same  decisions  if  they  used  only  the  two  factors  of 
information  indicated., 

This  suggests  that  these  two  factors  are  likely  to  be  most  suitable 
for  consideration  in  dealing  with  the  problem  of  establishing  a basis  for 
differential  classification  of  FCIC  State  Director  positionso  While  the 
finding  of  this  research  does  not  define  the  standards  for  such  classifi- 
cation, they  do  point  to  the  kinds  of  information  that  nrobably  would  be 
most  useful  in  developing  such  standards. 
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Exhibit  2 

Comparison  of  Rankings  Obtained  from  Weighted 
Factors  with  Average  Score  Rank 


State 

Identification 

1 (a) 

2 (b) 

We  ignited 

Rank 

Avo  Score 
Rank 

a 

3 

13 

10o5 

llo5 

b 

23 

23 

23 

19 

c 

10 

19 

16  0 5 

21 

d 

7 

3 

4 

2o5 

e 

16 

17 

18 

21 

f 

19 

11 

13 

16 

g 

20o5 

22 

22 

21 

h 

18 

15 

16o5 

9 

i 

20o5 

18 

19 

23 

13 

6 

6 

5 

k 

11 

9 

10o5 

14 

1 

17 

21 

20 

17o5 

m 

18 

12 

14 

14 

n 

22 

20 

21 

17o5 

0 

1 

14 

10o5 

14 

P 

2 

5 

3 

6 

q 

6 

10 

7 

9 

r 

9 

4 

5 

4 

s 

5 

2 

2 

1 

t 

14 

16 

15 

9 

u 

4 

1 

1 

2.5 

V 

15 

$ 

10o5 

7 

w 

12 

8 

8 

llc5 

Notes 


(a)  Ratio  of  total  acres  insurer  in  the  state  in  existing 
counties  to  total  insurable  acres  in  existing  counties 

(b)  Dollar  amount  of  premium  income 
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APPENDIX  A 


Ixistructions  to  the  Participant 


You  have  been  given  2 stacks  of  23  sheetSo  On  each  sheet  are  listed 
the  data  for  11  variables  that  we  agreed  to  include  in  the  experiment o 
Each  sheet  represents  one  state  director  posit ion o 

The  experiment  is  divided  into  two  phaseso  In  each  phase  you  will 
sort  the  sheets  into  3 piles  as  follows? 

High  - 7 sheets 

Medium  = 9 sheets 

Low  = 7 sheets 


The  way  to  do  this  is  to  first  go  through  the  stack  and  pick  out  the 
7 highest  sheets  and  set  them  aside o Then  go  through  the  stack  and  pick 
out  the  7 lowest  sheet So  This  will  leave  9 remaining  sheets  in  the  middle 
or  medium  pilso 

The  basis  for  sorting  in  each  phase  will  be  as  follows? 

Pha^e  1 = Sorting  exclusively  on  the  basis  of  the  relative  level  of 
the  joBo  In  making  these  judgments  think  of  the  position  as  being  unoc= 
cupiedo  Otherwise  you  may  take  the  variables  into  account  in  any  way  that 
you  see  fit  to  arrive  at  your  judgment  of  the  level  at  which  the  position 
should  be  put^  but  you  must  conform  to  the  distribution  establishedo 

When  you  complete  Phase  1 circle  the  Ls  for  the  low  pile^  the  Ms  for 
the  medium  pile^  and  the  Hs  for  the  high  pileo  Then  assemble  the  piles 
so  that  Low  is  on  the  bottom ^ Medium  in  the  middle  and  High  on  the  top  a 
Put  the  cover  sheet  on  the  stack  and  initial  it^  then  proceed  to  Phase  2o 

Phase  2 = Sorting  on  the  basis  of  the  relative  level  of  the  job  as 
perfo^eT~by  the  present  incumbent o In  making  these  judgments  take  into 
account  8 as  you  deem  appropriate s,  the  performance  of  the  present  incumbent o 

When  you  complete  Phase  2 circle  the  Ls  for  the  low  pile,  the  Ms  for 
the  medium  pile  and  the  Hs  for  the  high  pileo  Then  assemble  the  piles  so 
that  Low  is  on  the  bottom.  Medium  in  the  middle  and  High  on  the  topo  Put 
the  cover  sheet  on  the  stack  and  initial  ito 


APPENDIX  B 


Computational  Rationale  and  Variance 
Components  for  Intraclass  Correlation 


' The  computational  rationale  for  this  procedure  is  taken  from  Ebelg  R,  L. 

Estimation  of  the  Reliability  of  Ratings,  Psychometrika , 16(4),  December,  1951, 
‘ pp,  407-424  as  follows: 

Analysis  of  Variance  for  Level  (L) 


Source 

Degrees  of 
Freedom 

Sums  of 
Squares 

Mean 

Square 

Components 
of  Variance 

State 

director  positions  (p) 

22 

82,2 

3,74 

2 o 2 

a + 9 a 
e p 

Error 

(e) 

184 

43,8 

,24 

a2 

e 

Total 

206 

126 

, , 3,74  - ,24 

Intraclass  r = 

= ,62 

3.74  + 8(,24) 


Analysis  of  Variance  for  Level  Plus  Performance  (L+P) 


Degrees  of 

Sums  of 

Mean 

Components 

Source 

Freedom 

Squares 

Square 

of  Variance 

State  director  positions  (p) 

22 

94,89 

4,31 

a 

Error  (e) 

184 

31,11 

,17 

1 

Total 

206 

126 

Intraclass 


r = ^»31  - .17 
4.31  + 8(,17) 


.73 
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APPENDIX  D 


The  Utilization  of  Information  in  Multiple  Regression  Analysis 


The  manner  in  which  this  mathematical  technique  utilizes  information 
is  best  illustrated  by  a few  simple  diagrams o 

Suppose  that  we  have  two  categories  of  information  that  are  of 
interest  to  uso  These  can  be  represented  by  means  of  two  overlapping 
circles  or  sets  labeled  A and  Bo  The  area  of  overlap ^ the  shaded  area, 
represents  that  information  which  is  common  to  both  A and  B« 


Suppose  we  wanted  to  add  up  everything  in  A and  B by  adding  first 
everything  in  A and  then  everything  in  Bo  In  doing  this  we  would  have 
counted  everything  in  the  shaded  area  twice,  once  when  counting  A and 
once  when  counting  B®  In  order  to  be  correct  in  our  arithmetic  we  only 
want  to  count  the  shaded  area  once,  hence  we  would  subtract  it  once 
(A  + B = AtB  - <^)o 

Now  let  us  complicate  the  picture  a little  by  introducing  a third 
set  Co 


and  let  us  define  the  following  areas? 

- that  information  in  the  intersection  of  all  three  sets; 

- that  information  common  to  B and  C which  A does  not  have; 

///  - that  information  common  to  A and  C which  B does  not  have; 

- that  information  common  to  A and  B which  C does  not  have® 
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If  we  wanted  to  add  up  everything  in  all  three  sets 9 A + B r we 
would  end  up  counting  the  dotted  and  slanted  areas  twice  each  and  the 
solid  area  three  times o To  correct  for  this  we  would  subtract  the 
dotted  and  slanted  areas  once  each  and  the  solid  area  twice 9 io^o 
AtBtC  = ^-  (^  = <2;3><=.  2^0 

VJe  could  make  the  picture  increasingly  complex  by  adding  more  sets 
of  informatioRg  however^  this  example  is  sufficient o Let  us  ascribe  some 
meaning  to  these  categories  of  informat ion o 

Let  us  now  assume  that  A 9 Bg  and  C are  three  measures  on  a group 
of  people  1 

A = Score  for  each  person  representing  his  qualifications; 

B = Score  for  each  person  representing  his  characteristics; 

C ^ Score  for  each  person  indicating  his  **Readiness  for  Promotion 0” 

In  this  context  multiple  regression  analysis  would  attempt  to 
ascertain  the  manner  in  which  categories  A and  B are  combined  to  arrive 
at  an  individuates  ^‘Readiness  for  Promotion”  scoreo  The  outcome  would 
be  a set  of  weights  assigned  to  each  category  of  information  so  that 
when  combined  they  provide  the  "Readiness  for  Promotion”  scoreo  The 
weights  are  assigned  to  these  categories  so  as  to  maximize  their  rela= 
tionship  with  the  "Readiness  for  Promotion”  score  and  minimize  the  amount 
of  overlap  betvmen  categories  A and  Bo  ThuSj  in  the  following  examples 


The  solid  area  would  be  counted  as  part  of  A and  then  in  assigning 
a weight  to  B the  solid  area  would  not  be  counted  again o Thus,  B would 
play  a much  smaller  part  in  determining  C than  would  Ao 

An  individual's  A score  would  then  be  multiplied  by  the  weight  for 
A 9 his  B score  by  the  weight  for  Bg  and  then  these  two  weighted  scores 
would  be  summed  in  order  to  arrive  at  an  estimated  C scoreo  A measure 
of  the  adequacy  with  wnich  the  C scores  are  estimated  by  this  weighting 
system  is  called  the  multiple  correlation o This  value  varies  from  loOO 
for  perfect  estimation  to  oOO  for  lack  of  any  relationshipo 
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